Mipmapping has some unexpected performance characteristics. A texture with a full mipmap pyramid will take up ~33% more space than just the base level. So there is some memory overhead. The unexpected part is that this is actually a memory vs. speed tradeoff, as mipmapping usually improves performance.

If a texture is going to be minified significantly, providing mipmaps is a performance benefit. The reason is this: for a highly minified texture, the texture accesses for adjacent fragment shaders will be very far apart. Texture sampling units like texture access patterns where there is a high degree of locality, where adjacent fragment shaders access texels that are very near one another. The farther apart they are, the less useful the optimizations in the texture samplers are. Indeed, if they are far enough apart, those optimizations start becoming performance penalties.

Textures that are used as lookup tables should generally not use mipmaps. But other kinds of textures, like those that provide surface details, can and should where reasonable.

While mipmapping is free, linear mipmap filtering, GL_LINEAR_MIPMAP_LINEAR, is generally not free. But the cost of it is rather small these days. For those textures where mipmap interpolation makes sense, it should be used.

Anisotropic filtering is even more costly, as one might expect. After all, it means taking more texture samples to cover a particular texture area. However, anisotropic filtering is almost always implemented adaptively. This means that it will only take extra samples for fragments where it detects that this is necessary. And it will only take enough samples to fill out the area, up to the maximum the user provides of course. Therefore, turning on anisotropic filtering, even just 2x or 4x, only hurts for the fragments that need it.